Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloaders for historical transfers #1473

Merged
merged 36 commits into from
Jun 13, 2019
Merged

Downloaders for historical transfers #1473

merged 36 commits into from
Jun 13, 2019

Conversation

dshulyak
Copy link
Contributor

@dshulyak dshulyak commented May 29, 2019

developed on top of #1467

todo:

  • better algorithm for finding eth transfers

@status-github-bot
Copy link

Pull Request Checklist

  • Have you updated the documentation, if impacted (e.g. docs.status.im)?

@status-im-auto
Copy link
Member

status-im-auto commented May 29, 2019

Jenkins Builds

Click to see older builds (42)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 1d532a1 #1 2019-05-29 11:03:30 ~52 sec linux 📦 zip
✔️ 1d532a1 #1 2019-05-29 11:06:18 ~3 min ios 📦 zip
✔️ 1d532a1 #1 2019-05-29 11:10:00 ~7 min android 📦 aar
✔️ 84fcc96 #2 2019-05-30 07:47:14 ~57 sec linux 📦 zip
✔️ 84fcc96 #2 2019-05-30 07:48:56 ~2 min ios 📦 zip
✔️ 84fcc96 #2 2019-05-30 07:51:49 ~5 min android 📦 aar
✔️ a755abe #3 2019-05-30 08:02:07 ~37 sec linux 📦 zip
✔️ a755abe #3 2019-05-30 08:04:20 ~2 min ios 📦 zip
✔️ a755abe #3 2019-05-30 08:09:09 ~7 min android 📦 aar
✔️ 654f1bf #4 2019-05-31 06:43:11 ~59 sec linux 📦 zip
✔️ 654f1bf #4 2019-05-31 06:45:31 ~3 min ios 📦 zip
✔️ 654f1bf #4 2019-05-31 06:50:36 ~8 min android 📦 aar
✔️ 84fea8b #5 2019-05-31 08:21:16 ~2 min ios 📦 zip
✔️ 84fea8b #5 2019-05-31 08:23:41 ~40 sec linux 📦 zip
✔️ 84fea8b #5 2019-05-31 08:24:22 ~5 min android 📦 aar
✔️ f671872 #6 2019-06-01 06:15:46 ~41 sec linux 📦 zip
✔️ f671872 #6 2019-06-01 06:17:58 ~2 min ios 📦 zip
✔️ f671872 #6 2019-06-01 06:21:51 ~6 min android 📦 aar
✔️ 5e384d2 #7 2019-06-01 06:33:36 ~33 sec linux 📦 zip
✔️ 5e384d2 #7 2019-06-01 06:36:04 ~3 min ios 📦 zip
✔️ 5e384d2 #7 2019-06-01 06:38:52 ~5 min android 📦 aar
✔️ ec95d81 #8 2019-06-01 06:37:45 ~30 sec linux 📦 zip
✔️ ec95d81 #8 2019-06-01 06:40:20 ~3 min ios 📦 zip
✔️ ec95d81 #8 2019-06-01 06:45:39 ~6 min android 📦 aar
✔️ 4c5b957 #9 2019-06-01 06:57:28 ~35 sec linux 📦 zip
✔️ 4c5b957 #9 2019-06-01 07:00:54 ~4 min ios 📦 zip
✔️ 4c5b957 #9 2019-06-01 07:02:47 ~5 min android 📦 aar
✔️ 6a67f43 #10 2019-06-01 06:58:24 ~38 sec linux 📦 zip
✔️ 6a67f43 #10 2019-06-01 07:05:18 ~4 min ios 📦 zip
✔️ 6a67f43 #10 2019-06-01 07:08:24 ~5 min android 📦 aar
✔️ a383e85 #11 2019-06-01 07:04:16 ~32 sec linux 📦 zip
✔️ a383e85 #11 2019-06-01 07:08:05 ~2 min ios 📦 zip
✔️ a383e85 #11 2019-06-01 07:14:46 ~6 min android 📦 aar
✔️ 1fb7fc2 #12 2019-06-01 07:20:27 ~33 sec linux 📦 zip
✔️ 1fb7fc2 #12 2019-06-01 07:22:55 ~3 min ios 📦 zip
✔️ 1fb7fc2 #12 2019-06-01 07:26:46 ~6 min android 📦 aar
✔️ 8f5e869 #13 2019-06-02 06:22:00 ~1 min linux 📦 zip
✔️ 8f5e869 #13 2019-06-02 06:25:21 ~4 min ios 📦 zip
✔️ 8f5e869 #13 2019-06-02 06:28:35 ~7 min android 📦 aar
✔️ 7dfb23d #14 2019-06-03 08:26:31 ~32 sec linux 📦 zip
✔️ 7dfb23d #14 2019-06-03 08:29:22 ~3 min ios 📦 zip
✔️ 7dfb23d #14 2019-06-03 08:32:57 ~7 min android 📦 aar
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 45fbd2e #15 2019-06-03 08:28:34 ~33 sec linux 📦 zip
✔️ 45fbd2e #15 2019-06-03 08:33:14 ~3 min ios 📦 zip
✔️ 45fbd2e #15 2019-06-03 08:38:55 ~5 min android 📦 aar
4f02424 #16 2019-06-13 06:27:07 ~11 sec android 📄 log
4f02424 #16 2019-06-13 06:27:08 ~9.8 sec linux 📄 log
4f02424 #16 2019-06-13 06:27:08 ~11 sec ios 📄 log

@dshulyak
Copy link
Contributor Author

@yenda you can take this branch instead of #1467. it will download all historical transfers. for erc20 i am using getLogs for range of blocks. and eth is downloaded by comparing balance between ranges of blocks. so eth downloader might be a bit slow.

@dshulyak
Copy link
Contributor Author

@pombeirp @adambabik since you guys already reviewed previous PR you can help me with this one :)

@dshulyak dshulyak changed the title [WIP] Implement downloaders for historical transfers Implement downloaders for historical transfers May 31, 2019
@dshulyak
Copy link
Contributor Author

will implement better algorithm for finding eth transfers in a followup PR

@yenda
Copy link
Contributor

yenda commented May 31, 2019

@dshulyak
Copy link
Contributor Author

dshulyak commented Jun 1, 2019

there is a bug in ethclient https://github.com/ethereum/go-ethereum/blob/master/ethclient/ethclient.go#L243-L251 it doesn't check an error. i pushed a workaround and will submit a fix for it

@dshulyak
Copy link
Contributor Author

dshulyak commented Jun 1, 2019

made a fix here ethereum/go-ethereum#19653

@yenda btw can you give me an address that you are using? i want to run it with my script

@yenda
Copy link
Contributor

yenda commented Jun 1, 2019

@dshulyak 0x5DC6108dC6296b052BBd33000553AfE0Ea576b5E

@yenda
Copy link
Contributor

yenda commented Jun 1, 2019

I am not sure how to use wallet_getTransfers
In the json-rpc api block numbers are passed in hex, here it seems like it takes an integer

If I pass [7773254 nil] I get nothing, is it because the range start is the most recent block and end the oldest in the api? it is a bit confusing so the doc could be clarified on that
If I pass [nil nil] the app crashes https://gist.github.com/yenda/6c0ce03c321c05bc3ba354ef01e84248

This is all the params I tried with 0x5dc6108dc6296b052bbd33000553afe0ea576b5e:

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [783254 7773254]})
nil

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [773254 7873254]})
nil

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [7773254 7873254]})
nil

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [7773254 nil]})
nilnil

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [7873254 nil]})
nil

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [7873346 nil]})
nil

status-im.ethereum.json-rpc> (call {:method "wallet_getTransfers" :on-success println :params [nil nil]})
crash

@dshulyak
Copy link
Contributor Author

dshulyak commented Jun 2, 2019

If I pass [7773254 nil] I get nothing, is it because the range start is the most recent block and end the oldest in the api? it is a bit confusing so the doc could be clarified on that

it gets all transfers from first block till the last if last is not provided query will have no limit. query works in forward order if second blocks is lower that the first probably nothing will be selected

as for the first one is nil i will handle that.

have you tried to query using 0 as a first parameter?

In the json-rpc api block numbers are passed in hex, here it seems like it takes an integer

i will change it to hex

@dshulyak
Copy link
Contributor Author

dshulyak commented Jun 2, 2019

examples:

{"jsonrpc":"2.0","id":14,"method":"wallet_getTransfers","params":[0,20]}
{"jsonrpc":"2.0","id":14,"method":"wallet_getTransfers","params":[0,null]}
{"jsonrpc":"2.0","id":13,"method":"wallet_getTransfers","params":[0]}

@dshulyak
Copy link
Contributor Author

dshulyak commented Jun 3, 2019

changes in last commit:

  • More debug info when node is stopped
  • low keys in events
  • send history event only if new transfers were found

can you turn on debug logging for status-go? i wasn't able to repro 1st issue. but added couple of enhancements and logs when service is stopped

@dshulyak dshulyak changed the title Implement downloaders for historical transfers Downloaders for historical transfers Jun 3, 2019
When downloader is started it will go through following procedure:
- verify that the last synced block is still in canonical chain
- run fast indexer for every account from last synced block to head of the chain
- start watching block by block
@dshulyak dshulyak merged commit 5c7dbf3 into wallet Jun 13, 2019
@delete-merged-branch delete-merged-branch bot deleted the historical-transfers branch June 13, 2019 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants